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Exploration of Advanced Probabilistic and Stochastic Design Methods 


Executive Summary 

The primary objective of the three year research effort was to explore advanced, non-deterministic 
aerospace system design methods that may have relevance to designers and analysts. The research pursued emerging 
areas in design methodology and leverage current fundamental research in the area of design decision-making, 
probabilistic modeling, and optimization. The specific focus of the three year investigation was oriented toward 
methods to identify and analyze emerging aircraft technologies in a consistent and complete manner, and to explore 
means to make optimal decisions based on this knowledge in a probabilistic environment. 

The research efforts were classified into two main areas. First, Task A of the grant has had the objective of 
conducting research into the relative merits of possible approaches that account for both multiple criteria and 
uncertainty in design decision-making. In particular, in the final year of research, the focus was on the comparison 
and contrasting between three methods researched. Specifically, these three are the Joint Probabilistic Decision- 
Making (JPDM) technique, Physical Programming, and Dempster- Shafer (D-S) theory. 

The next element of the research, as contained in Task B, was focused upon exploration of the Technology 
Identification, Evaluation, and Selection (TIES) methodology developed at ASDL, especially with regards to 
identification of research needs in the baseline method through implementation exercises. The end result of Task B 
was the documentation of the evolution of the method with time and a technology transfer to the sponsor regarding 
the method, such that an initial capability for execution could be obtained by the sponsor. Specifically, the results of 
year 3 efforts were the creation of a detailed tutorial for implementing the TIES method. Within the tutorial package, 
templates and detailed examples were created for learning and understanding the details of each step. 

For both research tasks, sample files and tutorials are attached in electronic form with the enclosed CD. 
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Introduction 

The primary objective of the proposed research is to explore advanced, non-deterministic 
aerospace system design methods that may have relevance to designers and analysts. The research pursues 
emerging areas in design methodology and leverage current fundamental research in the area of design 
decision-making, probabilistic modeling, and optimization. The specific focus of the proposed investigation 
is oriented toward methods to identify and analyze emerging aircraft technologies in a consistent and 
complete manner, and to explore means to make optimal decisions based on this knowledge in a 
probabilistic environment. 

The objective in Task A is to arrive at formulations that allow for na n-dimensional examination 
of a probabilistic objective function space. Such an ability is critical for decision-makers who must allocate 
resources in the presence of uncertainty. Task B is focused upon exploration of the Technology 
Identification, Evaluation, and Selection (TIES) methodology developed at ASDL, especially with regards 
to identification of research needs in the baseline method through implementation exercises. 

Task A: Review and Assessment for Approaches to Multi-criteria Optimization 
under Uncertainty 

Design is a decision making process focused upon determining, ultimately, preferred sets of 
design parameters. Among the decision-making techniques, traditional single criterion approaches fail to 
account for many real world systems that have multiple (possibly conflicting) criteria. Fortunately, there 
has been a growing number of multi-criteria approaches. However, many of these operate on deterministic 
information for the system and environment, and such information is not typically truly known with 
confidence at the conceptual or preliminary phases. In a typical case, then, assumptions are made and the 
decision-making process proceeds. This exposes the decision to risk as a result of unaccounted uncertainty 
in assumptions. Moreover, the use of the new technologies within a design problem adds additional 
technical and program uncertainty to the design process due to readiness or availability issues. Thus, the 
present research task, Task A of the grant, has had the objective of conducting research into the relative 
merits of possible approaches that account for both multiple criteria and uncertainty in design decision- 
making. In particular, in the final year of research for which this document represents a reporting, the focus 
was on the comparison and contrasting between three methods researched. 

The remainder of this report is structured as follows. First, a review of the three techniques studied 
will be presented, including mathematical formulation and an assessment of strengths and weaknesses. 
These three are the Joint Probabilistic Decision-Making (JPDM) technique, Physical Programming, and 
Dempster- Shafer (D-S) theory. The JPDM and D-S were described extensively in the Year 2 progress 
report through two P.hD. theses that were completed. Following the review, a concluding summary is 
offered that collects the pertinent information on each technique and presents it in the form of 
recommendations for use depending on a designer’s particular problem characteristics. Finally, an appendix 
is included that contains code, run-time instructions, and an example of executing the JPDM method. It is 
hoped that this appendix can facilitate the immediate use of the technology by NASA researchers. The 
doctoral thesis that was the culmination of JPDM research, delivered to NASA under this grant last year, 
can be referred to for answers to detailed questions. 

1. Joint Probability Decision Making (JPDM) Technique 

The Joint Probabilistic Decision Making (JPDM) technique, which was developed at Aerospace 
System Design Faboratory (ASDF) and reported to NASA in last year’s report under this grant, 
incorporates a multi-criteria and a probabilistic approach to systems design. Its essential function is to 
estimate the probability of satisfying the criteria concurrently under uncertainty. This is done primarily 
through the construction of joint probability functions, which result when multiple random variables, each 
having their own marginal distribution, are involved. In JPDM, the criteria themselves are the random 
variables and so the joint distribution sought includes all of them. 

There are two models in the JPDM technique: the Joint Probability Model (JPM) and Empirical 
Distribution Function (EDF). JMP is an analytical approach while the EDF relies on empirical data to build 
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the joint probability mass function. Eq. (1) presents the joint probability density function / for JPM and Eq. 
(2) gives joint probability mass function for EDF model. 


f(x 1 ,x 2 ,....,x N ) = ^f Xi ■fx 2 "-fx N 


with C = J---|/ X; -f X] ■■■/ Xn ■g(x 1 ,x 2 ,....,x N )dx I dx 2 ---dx N 

Q 

fxi is marginal distribution function, and g(x 1 ,x 2 , ...,x N ) is correlation function 


M 

f Zl Z 2 Z 3 ( a ’ b ’A = M S 7 ( Z li = a ’ Z 2 / = b ’ Z 3 i = C ) 


i = 1 


where 


(i) 


( 2 ) 


I(z n =a, z 2i -b,z 3j =c)={ 


1 if true 
0 if false 


JPDM uses Probability of Success (POS) as the ultimate measure of optimality and this is given 
by Eq. (3) from [Ref. 1]. Criteria j and Criteria 2 are two system objectives that are investigated Constrain i 
and Constrain 2 are limitation for the two criteria and construct the area of interest 

POS = P(Criteria j e Constrain n Ctiteria 2 e Constrain 2 ) (3) 

The two-dimensional representation of JPDM environment is shown in Figure 1. The fact that 
JPDM is a multi-critera design technique with a probabilistic theme is clear in the figure. One of its 
strongest advantages is that it captures the uncertainties directly by applying the probability distributions to 
the input design parameters which result in the criteria being random variables. In Figure 1, the marginal 
distributions for the criteria are shown in blue along the axes. The POS, another advantage, is obtained by 
integrating the joint criterion distribution over the area of criterion values that are of interest to the 
customer. Finally, JPDM creates a compensatory technique for product selection (i.e. choosing the best 
amongst a finite set of alternatives) that allows comparison of alternatives on an equal basis. 




Figure 1 : JPDM technique visualization 
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JPDM also has some application limitations. When an analytical form of the criteria as a function 
of design variables is not available, empirical or simulation data must be used through the EDF approach. 
In the EDF model, accurate data is needed for the criteria. Such data is often not easy to get in conceptual 
and preliminary design phases. If an analytical expression is available and the JPM model is used, a 
correlation function is required, which also may not be available in early stages of design. 

The Procedure for Running JPDM Using EDF Model 

An example is presented to show how to use EDF model to conduct concept selection problem. 
The flow of data in this procedure is shown on Figure 2. This example is also relevant for this grant since it 
was connected to other work on-going in the NASA Systems Analysis Branch in the area of Personal Air 
Vehicles (PAV) under the RASC program. In this example, it is desired to select the most viable PAV 
concept from amongst three competing concepts: a version of the Robinson R-44 helicopter, a version of 
the Groen Brothers Aviation Hawk4 gyroplane, and the Bell/ Augusta 609 tiltrotor. The actual code for 
executing this problem using JPDM is described in the Appendix to this report. 

© DOE 



probtable = 

0 0 1 2 

1 0.3228 0.3305 0.8757 

2 0.1149 0.1202 0.8125 

3 0 0 1.0000 




Figure 2: Flow chart of product selection procedure using EDF model 
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Identify the Criteria and Their Range of Interest 

Before running JPDM, the criteria which evaluate the customer requirements should be identified 
first. In the example of PAV concept selection problem, three criteria are identified. They are doorstep to 
destination (D-D) time, direct operating cost (DOC) and noise (NOISE). Viability of a design concept is 
measured by the probability of satisfying certain desired levels of the doorstep to destination time, direct 
operating cost and noise. All the criteria are desired to be as small as possible, so zero as a minimum value 
is assigned to all the criteria. Upper limits that cannot be violated are identified as 3.5 hrs, 130 $/hr, 79 db 
for doorstep to destination time, DOC and noise respectively. These values create the constraints for the 
three criteria: 0 < D-D < 3.5 hr, 0 < DOC < 130 $/hr and 0 < NOISE < 79 db. 

Obtain Empirical Data 

Empirical data can be obtained from different sources. In this example, data were generated with 
the combination of aircraft design codes, Response Surface Equations (RSEs), and the Monte Carlo 
Simulation and then used for the EDF. An RSE is a second-order quadratic equation developed from the 
Response Surface Methodology (RSM) multivariate regression technique to model the responses of a 
complex system. Before RSEs are constructed, modeling and simulation is conducted using sizing and 
performance codes relevant to each vehicle such as GTPDP (helicopters, [Ref. 2]) and VASCOMP 
(tiltrotor, [Ref. 3]). The outcomes serve as the regression data guided intelligently through the Design of 
Experiment (DoE) technique. Once the response data is collected, RSEs can be generated using a statistical 
analysis package called JMP [Ref. 4]. When RSEs are developed, 10,000 samples are generated through the 
Monte Carlo Simulation (MCS) using Crystal Ball software. In the phase of MCS, uncertainty is 
propagated to the system level by defining appropriate probability distributions to uncertain mission 
requirements, vehicle attributes and infused technology. The sample data for each concept is saved as 
Datal Data2 and Data 3. The data files have to contain the data in an m-n matrix, m being the number of 
rows, i.e. samples, n being the number of columns, i.e. criterion. The number used after “Data” corresponds 
to the concept to be evaluated. 

Run EDF Model 

The JPDM was coded using MATLAB and FORTRAN. The flow of information within the 
program is outlined in the flow chart of Figure 3 [Ref. 1]. Each of these files is also contained on the CD 
supplied with this report. The values of some parameters should be modified before the program can work 
for the specific problem. In the inputsEDF.m file, the elements in the matrices called maxvalue and 
minvalue should be changed to the upper and lower limits of the criteria identified before. Thus, for this 
specific case, maxvalue=[130 79 3.5] and minvalue=[0 0 0]. It should be noted that the order of the 
criteria in the data file and the order of the criteria in the above two matrices should be identical. In other 
words, if the first column in the data files stands for criterion DOC, the first elements in the matrices 
maxvalue and minvalue have to be the upper and lower values of DOC. The numbers in the matrix crplots 
indicates which two criteria will be evaluated for calculating the Probability of Success and creating the 
graphs. For example, if crplots= [1 2 ] the POS and graphs will be generated for the criterion 1 and 2 
(DOC vs. NOISE). In addition, name(l) and name(2) should be changed to the appropriate name of the 
criteria that will be displayed in the graphs. 

After the above modifications are finished, the JPDM.m MATLAB file is run. The program will 
calculate the POS and create the graphs using EDF model. In this example, MATLAB gives the solution 
denoted in Table 1. The joint POS and univariate probabilities of for DOC, D-D and NOISE are shown for 
the three concepts. From a product selection point of view, the R-44 version has the greatest chance of 
success considering all three criteria simultaneously. Graphical representation of NOISE-DOC slice is 
shown in Figure 4. Similarly, the other two two-dimensional graphs (DOC vs. D-D and NOISE vs. D-D) 
can be obtained by changing the parameters in the matrix crplots. 
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Figure 3: Flowchart of the Joint Probabilistic Decision Making Technique 
Table I: Example JPDM Results- POS and Univariate Distributions 



Joint Probability of 
Success 

Univariate 
Probability for 
Criteria DOC 

Univariate 
Probability for 
Criteria NOISE 

Univariate 
Probability for 
Criteria D-D 

R-44 

0.1448 

0.3305 

0.8757 

0.4378 

HAWK 4 

0.0689 

0.1202 

0.8125 

0.3716 

B/A-609 

0 

0 

1.0000 

0.9992 



Figure 4: Joint Probability Distribution 
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2. Physical Programming (PP) 

Physical Programming (PP) is multi-criteria optimization method that requests physically 
motivated information from the designer and produces a problem statement that reflects the realistic texture 
of the designer’s preferences. It is described extensively in Refs. [5, 6, 7]. It seeks to avoid the traditional 
problem in the multiple criteria domain: the need to choose weightings for the criteria. In PP, after 
identifying the design parameters and design metrics, the designer specifies the class-type (Figure 5) for 
each metric and then also defines the ranges of different degrees of desirability for each design metric in 
the corresponding class function (Figure 6). Class types can be either soft or hard, depending on whether 
there is a crisp delineation between acceptable and unacceptable levels (hard) or a continuum of 
acceptability (soft). For example, Class 1-S indicates a metric for which smaller is better. Further, through 
the process in Figure 6, the degree to which levels of the metric are desirable can be defined. All soft class- 
function will become constituent components of the aggregate objective function (defined below in Eqs. 
(4)-(13)) that is minimized. The mapping which defines the path from this design parameters to aggregate 
objective function are shown in Figure 7. 



Figure 5: Classification of design metrics 
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Figure 6: Class-function ranges for the i-th generic metric 


Minimize: Z(x) : 


log 


10 


1 n sc 


gi(x) < gi5 

gi(x) > giS 

gi5L < gi(x) < g i5 R 


n i = i 

sc 

Subject to: 

(for class 1-S) 
(for class 2-S) 
(for class 3-S) 


(4) 


(5) 

( 6 ) 
(7) 
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gi5L < gi(x) < 

g i5R (for class 4-S) 

(8) 

gi(x) < giM 

(for class 1-H) 

(9) 

gi(x) > giM 

(for class 2-H) 

(10) 

§iM — Si.val 

(for class 3-H) 

(11) 

g iM < gi(x) < 

g iM (for class 4-H) 

(12) 

< Xj < XjM 

( for des. var. constraints) 

(13) 


where n sc = # of soft classes. 


Examples 

^ Design Var 

Length, x e 

Height 

Mapping 

iabte Space X 

R n 

i J 


Mapping depends of 

laws of nature, 

and human laws. 

r ^ 

/ Obgedh, 

Deflection, 

Cost \ 9 r - 

> 

f J 


Mapping depends of 
^ designer preference: 

preference ranges, 

and '■'■fgcgcig 

f* \ 

Is large for bad f 

design metric values Class-Fur 

is small for good ~~ 

design metric values, 9 = 

F “ A 

icdon Space 

f J 


Mapping is 
^ additive. 

Is targe for bad ^ 

overall design. f H 

ts small for good Aggregate Obi 

overall design. Sp 

V s eRl 

jechve Function 
aoe 

. PM J 


Figure 7: Physical Programming Mappings 

PP, as one of the multiobjective design methods, has some advantages over the conditional design 
methods. First, the formation of an aggregate objective function usually involves a largely ill-defined trial- 
and-error weight-tweaking process that may be a source of frustration and significant inefficiency. PP 
captures the designer’s physical understanding of the desired design outcome in forming the aggregate 
objective function, eliminating the uncertain and time-consuming process of scalar criteria weighting. 
Secondly, PP attempts to offer a problem formulation and solution framework that conforms to real-life 
design. The notion that attempts to use numerical weights to describe designer preference is considered as 
ineffective. PP does not require the designer to specify optimization weights in the problem formulation 
phase, and it allows designer to define their preference in physically meaningful terms. This removes the 
frustrating process of weight tweaking entirely. 
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PP also has its own drawback. Firstly, PP requires a priori selection of range parameters for each 
of the objective functions. Such information may not be known at the outset (e.g. “Is gross weight likely to 
range between 100,000-200,000 lbs in my design space or 200, 000-300, OOlbs?”). In problem formulation 
phase, the designer’s time is mostly consumed in exploring the implication of the various physical 
meaningful preference choices. Secondly, PP only provides information for one design scenario at a time. 
To capture a variety of design scenarios, which represent different preferences on the design metrics, a set 
of preference structures should be built and tested. Thirdly , PP is deterministic design method and does 
not capture the uncertainties due to incomplete information existing in design space and environment 
This latter issue is very significant. 


3. Dempster-Shafer (D-S) Theory 

Dempster- Shafer (D-S) theory of evidence (Ref. [8]) is a framework for integrating pieces of 
information from independent sources, in order to improve the decision making process. Unlike JPDM, D- 
S theory does not employ a probabilistic mentality. Instead, the concept of belief is used. Therefore, the D- 
S theory may be applied to integrate judgmental adjustments from several experts as required by the case at 
hand. The judgements for several candidate adjustments form basic belief assignments (BBA) over the set a 
Universe of Discourse U, also called Frame of Discernment, which is a set of mutually exclusive 
alternatives, in a way that the mass of belief assigned to the set varies from 0 (complete ignorance) to 1 
(perfect knowledge). The frame of discernment may consist of the possible values of an attribute. 

The basic belief assignment {BBA) is a mapping m from the set of all subsets A of U to [0, 1], such 

that 

m((|)) = 0 (14) 

2>(A) = 1 (15) 

AcQ 


Associated with m , the belief or credibility , and plausibility measures are defined by 

bel(A)=£m(B) (16) 

BcA 

pl(A) = J>(B) (17) 

AnB^(|) 


bel(S) summarizes all our reasons to believe A while pl(A) expresses how much we should believe 
in A if all currently unknown facts were to support A. Thus the bel(A) is not greater than pl(A) and true 
belief in A will be somewhere in the interval [bel(A), pl(A)]. 

Given two BBAs, mj and m 2 , the combined belief is computed using an important operation in the 
D-S theory. This operation is referred as “Dempster’s rule of combination”, which yields a new BBA 
m=ml+m2 defined as: 

2^ m 1 (B)m 2 (C) 

_ BnC=A ^ 

^m 1 (B)m 2 (C) ’ 

BnC*<|> 

The Dempster-Shafer theory can be used to represent situations in which different kinds of 
ignorance exist in our knowledge about a phenomenon or a system. Thus it is not tied to the need for input 
probability distributions 9as JPDM is). The belief functions are based on a basic probability assignment, 
which is a measure of the belief committed exactly to a given hypothesis or a subset of the frame of 
discernment. The theory has been applied to several decision making problems such as the selection of 
bearings for a specific application and for multiple objective optimization of engineering systems. Since 
most practical design problems are solved iteratively, the computational information accumulated in each 



(18) 
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iteration can be used to develop an evidence-based model using Dempster-Shafer theory for solving design 
problems more efficiently. 

This theory also has its own limitations. If the source of evidence is not independent, it can lead to 
misleading and counter-intuitive results. Secondly, in classical decision analysis, it remains necessary to 
enumerate the potential states of nature and to assign utilities. This task can be overwhelming when 
complex scenarios are considered. 


4. Summary Assessments 

From the analysis above, JPDM, PP and D-S theory are all multi-criteria design approaches. Only 
JPDM and D-S theory address uncertainty in the design problem in a direct way. PP allows the designer to 
express design preference in a flexible and physical meaningful manner, thus dealing with uncertainty in 
the formation of the objective function itself. In this way, PP eliminates the frustrating process of weight 
tweaking which the weight-based approaches have. This indicates PP is easier to use when weighting of 
criteria is difficult, risky, or not possible. 

JPDM has the ability to capture and analyze uncertainties, so it is good for the design problem 
with incomplete information. This is quite important in conceptual and preliminary aerospace vehicle 
design phases, when key information is known only to a partial extent. Further, in practice, more 
uncertainties exist in the operational environment of a vehicle as well as in computer model fidelity and 
technology insertion. Thus, JPDM is better than PP or D-S for practical design problem under explicit 
uncertain information. 

D-S theory is a generalization of probability theory with special advantages in its treatment of 
ambiguous data and the ignorance arising from it. This theory also has the ability to capture the 
uncertainties by using the belief functions that are based on a basic probability assignment. However, it 
computes the probability that evidence supports the proposition using belief intervals rather than computing 
the probability of a proposition. It can provide help for decision making when the probabilistic information 
is incomplete. 

According the analysis above, the three techniques and the associated problem characteristics for 
which they are well suitable to solve are shown in Figure 8. 



Figure 8: Comparison of Three Multi-criteria Decision-making Techniques 
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Task B: Development and Implementation of Technology Selection Methods 

The focus of Task B efforts for Year 3 was three fold, including: 

• Construct and document software implementation for the TIES method based on results of first 
and second year of research 

• Continue exploration of new methods, guided by Year 1 and 2 results in conjunction with SAB 
initiatives 

• Publish results of research in relevant conferences and journals 

The underlying theme of Year 3 efforts was to create an initial capability for the Systems Analysis 
Branch at NASA Langley to implement the TIES method. In effect, mature the method to the point that a 
technology transfer could occur so as to facilitate robust annual technology benefit assessments for the 
sponsor. To accomplish this end, ASDL implemented the TIES method into the Advanced Design Methods 
(class number AE8804) within the graduate program in the School of Aerospace Engineering at Georgia 
Tech. To test the robustness and ease of implementation, ASDL believed that if a group of first year 
graduate students with no knowledge of the techniques or analysis tools utilized in the TIES method could 
in fact leam and apply the method within a semester course, the method would be mature enough to 
transfer as a package to the sponsor. Given this goal, ASDL implemented the TIES method as a class 
project for the past 3 years with a culmination of all the method advancements included in the Fall 2002 
class. 

Within the graduate course, the 51 first year graduate students were divided into 17 teams of 3 
people. Each team was provided with a baseline aircraft model, the TIES tutorial, and a project description 
outlining the requirements for the project. The baseline aircraft models were obtained from the sponsor in 
the Summer of 2001 and corresponded to the fleet of aircraft being utilized for Enterprise technology 
benefit assessments. Additionally, the technologies utilized in the project were also provided (and 
consistent) with the sponsor assessments from 2001. Each team implemented the TIES process and was 
required to give a final presentation and report at the end of the semester. The team with the superior report 
is provided as an appendix, along with the tutorial, with the project description provided herein. 

Although the students were not experts in the techniques and software utilized in the TIES 
method, the final product delivered by the students indicates that the maturity of the TIES method is ready 
for transfer to the sponsor. The students implemented each step with minimal guidance and supervision. 
Given this result, a tutorial with sample files is also contained on the CD so as to facilitate the use of the 
TIES method within the Systems Analysis Branch. With this tutorial, the objectives of Year 3 were 
achieved. 
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AE 8804A: Advanced Design Methods I 
Fall 2002 Design Project Description 
Prof. Dimitri N[avr\s 
Asst Instructor: Dr. Michelle R. Kirby 


Important dates 

Deliverable 1 
Deliverable 2 
Deliverable 3 
Deliverable 4 
Deliverable 5 


September 27 
October 7 th 
October 23 rd 
November 13 th 


Detailed GANTT Chart of the Project: 

Steps 1-3 Report Due: 

Steps 4-5 Report Due: 

Steps 6-7 Report Due: 

Step 8 Report Due: December 6 th 

(Compiled in the Final Deliverable) 

Final Deliverable Report Due: Friday, December 6 th by noon to Dr. Kirby in both paper and 
electronic format (MS Word only). Each of the deliverables should be building 
toward this final report. 

Presentations: 40 minute presentation limit 

Teams will present their projects the week of December 2 nd . Details of the dates 
and time will be announced at a later date. 


With each report, the following is required and must be signed and dated by each team member: 

1 . A formal declaration that the work and ideas contained within the report are solely 
those of the team member except where explicitly referenced or cited from another 
source. 

2. A list of which team member performed a given requirement and who wrote a given 
section of the report. 

3. A table consisting of an evaluation of each team member to the team goals as 
evaluated by others. A sample will be provided as the first deliverable approaches. 


Important Information 

Website for tutorials, spreadsheets, project files, and sample scripts: 
http://www.asdl.gatech.edu/people/mkirby/Class/index.html 

A report format guideline will be posted on the class website shortly as well as a link to guidelines 
for writing a research paper and how to cite reference material. 


Objective 

The primary objective of the project is for you to learn the advanced design techniques taught in class. The 
focus of the project will be an exercise of the Technology Identification, Evaluation, and Selection (TIES) 
method. This mature method entails an eight-step process that you will implement for different aerospace 
vehicles. At the end of project, you should have learned the following: 

1 . How to run synthesis/sizing and economical analysis tools 

2. Design of Experiments and the usefulness of the method 

3. How to use the statistical package, JMP 

4. How to generate Response Surface Equations and check the goodness 

5. How to model uncertainty through Monte Carlo simulations 

6. Sensitivities of performance and economic metrics to design variables 
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7. Sensitivities of economic metrics to economic uncertainty 

8. How to simulate technology impacts on a vehicle 

9. How to choose the proper mix of technologies to meet customer requirements 

10. How to use and create UNIX shell scripts 

1 1 . How to write a research paper 

Each team is given a baseline vehicle which is modeled in the Flight Optimization System (FLOPS) code 
and the Aircraft Life Cycle Cost Analysis (ALCCA) accessible from class website. Each vehicle is 
representative of a current commercial vehicle or a notional future concept. You will be implementing all 
of the steps of the TIES method for the projects, in addition to expansions of the current method. Some 
information of particular steps will be provided to reduce the workload, specifically, the metrics, design 
variables of interest, scripts, and MS Excel® templetes. All other steps will be implemented by your team. 
References associated with the TIES method are listed below and can be obtained from the class website or 
from: http://www.asdl.gatech.edu/publications . 

1. Kirby, M.R., Mavris, D.N., "A Technique for Selecting Emerging Technologies for a Fleet of 
Commercial Aircraft to Maximize R&D Investment,” SAE-2001-01-3018. 

2. Kirby, M.R., Mavris, D.N., ”A Method for Technology Selection Based on Benefit, Available 
Schedule and Budget Resources,” SAE Paper No. 2000-01-5563. 

3. Kirby, M. R. and D. N. Mavris, ’’Forecasting Technology Uncertainty in Preliminary Aircraft 
Design,” SAE paper no. 1999-01-5631. 

4. Mavris, D.N., Kirby, M.R., ’’Technology Identification, Evaluation, and Selection for 
Commercial Transport Aircraft”, 58th Annual Conference Of Society of Allied Weight 
Engineers, San Jose, California 24-26 May, 1999. 

5. Mavris, D.N., Kirby, M.R., Qiu, S., ’’Technology Impact Forecasting for a High Speed Civil 
Transport”, SAE-985547. 

6. Mavris, D.N., Mantis, G., Kirby, M.R. ’’Demonstration of a Probabilistic Technique for the 
Determination of Economic Viability," SAE-975585. 

7. Kirby, M.R., Mavris, D.N., ”An Approach for the Intelligent Assessment of Future 
Technology Portfolios,” AIAA 2002-0515. 

The 8 steps contained in the TIES method are 

1 . Problem definition 

2. Baseline and alternative concepts identification 

3. Modeling and simulation 

4. Design space exploration 

5. Determination of system feasibility/viability: probability of success 

6. Specify Technology Alternatives 

7. Assess Technology Alternatives 

8. Select Best Family of Alternatives 

The requirements you will need to fulfill for the current project are listed as part of each step described 
below. Your effort in this project should not be limited to the pre-specified requirements. Original thought 
and creativity are desired and recommended. 


Step 1: problem definition 

The primary objective of Step 1 is to map the “voice of the customer” to the “voice of the engineer” as a 
response to some societal need as determined through a Quality Function Deployment as learned in the 
Systems Design for Affordability class. The result of this step is the identification of the system level 
metrics which capture the customer requirements. The metrics of interest are provided for you in Table II. 
As you can see, the target and/or constraint values are not provided and are to be determined (TBD) by 
your team. Suggestions for values include: NASA ( http ://www.hq .nasa. gov/office/aero /), Boeing and 
Airbus web sites for competitive economic values, Air Transportation Association web site 
( http ://www. airlines . org/public/ industry/ ), potential city pairs for runway lengths, FAA web site for 
operational restrictions, and so on. Also, the * metrics should have 2 targets. One for the year 2007 (10 
year goal) for a 25% reduction from your baseline levels, and the other for the year 2022 (25 year goal) for 
a 50% reduction from your baseline values. 
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Note, the following metrics are NOT calculated within FLOPS/ALCCA and must be calculated off-line and 
with the results from your analysis: 

Mission Block Fuel , Wing _wt 


COH ASM = 3.155- 


( Total _ Pax _ Capacity) * {Range) 


, and 


WAWt : 


Wing _ Area 


Requirements for Step 1 : 

1. Define the need for your aircraft, for example, projected passenger traffic and cargo growth, 
congested airways, or changing regulations. 

2. Define potential markets in which the aircraft will be utilized 

3 . Define competitors 

4. Define the metrics in Table II 

5. Describe why the metrics are important to your vehicle, what is the qualitative significance of 
lowering or increasing the metric values, i.e., what options or benefits does it provide to your 
vehicle. 

6. Who is the metric responding to, e.g., Acq $ is a metric of concern to the airlines. The more 
they pay for an aircraft, the larger the passenger ticket price needed to pay for the aircraft for 
the airline to make a particular profit, the more the investment the airlines must make, etc. 

7. Identify target or constraint values for each, suggestions for this are market studies by Boeing 
and Airbus, FAA regulations, airport compatibility, etc. You can also look at the baseline 
output file for starting values to get an estimate. An example for a target would be a TOFL of 
less than 11,000 ft for a HSCT. This is because most major airport runway lengths are less 
than 1 1 ,000 ft and the airplane must be able to take off without modifications to ANY existing 
airports. For a smaller capacity aircraft, being able to land or takeoff from more airports is 
important and a smaller runway length constraint would be important. Hint: you have a 50 
pax vehicle, optimizing utilization in different airports would be beneficial, think of PDK to 
Lake Charles, LA and find the associated runway lengths. 


Table II: System Level Metrics 


Parameter 

Target/ 

Constraint 

Performance 


Approach Speed (Vapp) 

TBD 

Landing Field Length (LdgFL) 

TBD 

Takeoff Field Length (TOFL) 

TBD 

C02/ASM (C02)* 

TBD 

NOx/ASM (NOxj* 

TBD 

Takeoff Gross Weight (TOGW) 

TBD 

Economics 


Acquisition Price (Acq S) 

TBD 

Research, Development, Testing, and Evaluation Costs 

(RDT&E) 

TBD 

Average Required Yield per Revenue Passenger Mile 

(S/RPM) 

TBD 

Total Airplane Related Operating Costs (TAROC) 

TBD 

Direct Operating Cost plus Interest (DOC+lj* 

TBD 

Misc 


Wing Aerial wt (WAWt) 

TBD 


Step 2: Baseline and alternative concepts identification 

Once the customer requirements are defined in terms of quantifiable engineering parameters, the thrust of 
the TIES method begins with the definition of the concept space. Initially, the experience, knowledge, and 
intuition of the designer is utilized to identify a potential class of vehicles and provides the methodology 
with a starting point for selecting potential solutions to satisfy the customer requirements. The focus of this 
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step is two-fold: identify the space of alternative concepts that is based on a defined class of vehicles, and 
establish the geometric and propulsive design space for which system feasibility is initially sought. 


Space of alternatives 

In the design of any complex system, there exists a plethora of combinations of particular subsystems or 
system characteristics that may satisfy the problem at hand. For example, how many engines are needed? 
What type of high lift system is needed? Is a horizontal stabilizer preferred over a canard? A functional and 
structured means of decomposing the system and identifying component options is through the use of a 
Morphological analysis. The Morphological Matrix is an ordered Morphological analysis that is formed by 
identifying the major functions or characteristics of a system on the vertical scale and all the possible 
alternatives (or system attributes) for satisfying the characteristics on the horizontal scale. In essence, this is 
where mature and immature technology alternatives are defined. Once the matrix is populated, an 
alternative design concept is defined as a mix of the characteristic alternatives. All possible design 
alternative combinations define the alternative concept space. In general, one alternative concept is 
established to begin the feasibility investigation and will be called the baseline concept and is typically 
drawn from mature or present day technologies. 


Design Space 

Once the baseline concept is defined from the alternative concept space, the baseline may be further 
decomposed into product and process characteristics. This can be performed via a Morphological Matrix or 
through brainstorming sessions. The metrics for your baseline are broken down into product characteristics 
(or design variables) in Table III. These “control” variables define the design space you will investigate for 
technical feasibility once you have identified appropriate ranges for each variable. The control variables 
are listed as the way they appear in FLOPS, the associated namelist, and a description of the variable. 

NOTE: the baseline files that you have in your accounts corresponds to a vehicle that could be built in 1997 
with the technologies available at that time. These vehicles are representative of the systems that NASA is 
considering in the Aerospace Technology Enterprise focus of the future. Please refer to NASA’s web site at 
http ://www.hq .nasa. gov/office/aero for more information. The technology sets and some objectives were 
founded based on the NASA efforts and the 10 and 25 years goals that were set forth by the NASA 
Administrator to guide and focus research endeavors of the future. This implies that your project is relevant 
to the research currently being conducted in the aerospace industry. 


Table III: Design Variable Description 


Variable 

Namelist 

Description 

SW 

CONFIN 

Wing area 

TWR 

CONFIX 

Thrust to weight ratio 

AR 

CONFIN 

Wing aspect ratio 

TR 

CONFIN 

Wing taper ratio 

TOC (1) 

WTIN 

Wing thickness-to-chord ratio at root 

TOC (3) 

WTIN 

Wing thickness-to-chord ratio at tip 

SWEEP 

WTIN 

Wing quarter-chord sweep 

ARHT 

WTIN 

HT aspect ratio 

TRHT 

WTIN 

HT taper ratio 

TCHT 

WTIN 

HT thickness-to-chord ratio 

SHT 

WTIN 

HT area 

ARVT 

WTIN 

VT aspect ratio 

TRVT 

WTIN 

VT taper ratio 

TCVT 

WTIN 

VT thickness-to-chord ratio 

SVT 

WTIN 

VT area 
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Requirements for Step 2: 

1. Describe your baseline and identify what class of vehicle it is, i.e., long range, widebody, etc 
and provide a notational 3 view. Create a Morphological Matrix of your vehicle based on the 
class of vehicle that it falls into. You will need two major components of your matrix 
including mission requirements and vehicle characteristics. Once you create the Matrix, 
please identify the baseline as a picture with the baseline characteristics circle. Be as 
exhaustive as possible and note that there is no single or right Morphological Matrix. This is a 
tools for brainstorming. 

2. Define “workable” ranges for the design variables in Table III, where workable implies that 
FLOPS can achieve a converged solution for any combination of variables. Note, this is 
probably the most time consuming and difficult aspect of the project. You will need patience. 
© Suggestions would be a deviation of +/- 20-30% from your baseline values if you cannot 
find any information. Note, FLOPS is HIGHLY sensitive to variations in T/W and W/S ratios, 
especially T/W. You may not want to deviate as much as 20-30 %. Also, consider which of 
the design variables provided will have the most significant impact on the system and use 
those as drivers to determine your ranges. 

Step 3: Modeling and simulation 

The modeling and simulation environment is provided to you as the FLOPS and ALCCA executable. All 
teams are provided a FLOPS Version 5.95 and an ALCCA Version 5 manual. I would strongly suggest 
understanding ALL the inputs that describe your vehicle and the inputs that are omitted (set at default 
values). You should play with the program a bit to get a feel for how it works. O Note, you should make a 
backup of all the files that are provided to you before you make ANY modifications to the baseline files. 


Requirements for Step 3: 

1 . Describe the mission that is used to size your vehicle. 

2. What are the different segments of the mission trying to optimize and/or how is the aircraft 
operating for a given segment. For example, you could climb for minimum fuel bum or 
minimum time. Specify each segment or what power setting are you using for approach. 

3. What are the assumptions associated with sizing and evaluating your vehicle, both from a 
sizing perspective and an economic perspective? 

4. Provide a figure representing your mission profile 

5. Plot your engine performance deck in the following formats 

1 . Thmst v. Mach for different altitudes 

2. Fuel flow v. Mach for different altitudes 
Do both of these at max power 

6. Plot your baseline drag polars for different Mach numbers at sea level and at your cmise 
altitude. (You can find this information in your baseline output file.) 

7. From your baseline file, establish the following: 

A) Airframe manufacturer related information 

• What is the manufacturer’s cash flow for your vehicle? 

• What are the elements composing the cash flow? 

• What is the breakeven unit number and when in the program does it occur? 

• Show the sensitivity of the manufacturer’s ROI as a function of Aircraft Price for 
different production quantities. 

• What is the production schedule? That is, how many aircraft are being produced per 
month and for how many years? 

• What is the acquisition cost of the aircraft as a function of units (in logi 0 scale) 
produced? This should be a straight line. 

• Vary the production schedule (ie, number of years and how many per month) and 
show and discuss the impact on the manufacturer’s cash flow, the breakeven unit and 
time, and the acquisition price as a function of units produced. 

B) Airline related information 
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• Provide a cost breakdown of the economics of your aircraft for the design and 
economic range. For example, a pie chart of the contributing elements of the TOC, 
DOC, and IOC. 

• Show the sensitivity of the Airlines ROI as a function of Aircraft Price for different 
average yields. 

• Show the ROI for the operation of your vehicle throughout it’s life, that is, how do 
the following vary with years of operation: operating cost, interest, depreciation, 
earnings before tax, net earnings, net cash flow, and discounted cash flow. Define 
each of these terms. 

Step 4: Design space exploration 

Before you begin the design space exploration, establish datum values for all the metrics listed in Table II. 

You can do this simply by looking at your baseline output file provided to you in your team account. You 

will be using the metamodel and Monte Carlo simulation method for the design space exploration. 


Requirements for Step 4: 

1 . Describe what a Design of Experiments is. Why are you using Design of Experiments 

2. What is an R 2 , a whole model test, a residual 

3 . Describe how to generate a Response Surface Equation (RSE) 

4. Create RSEs of the metrics and describe which DoE you used 

5. Create prediction profiles of the metrics to show the sensitivity of the response to the design 
parameters. Discuss the results and trends. Do they make sense 

6. Desribe the different means to evaluate the accuracy and goodness of your RSEs. Select a 
couple of approaches and show how accurate the RSEs are by providing the confidence 
intervals and error distributions and a description of how you came up with them. Run a 
confirmation test 

7. You must run as many randomly generated cases as you had for your original DoE. Dr. Kirby 
has files that will set these up for you. Just email or go see her. Once you run these cases, 
import them into an MS Excel® file. Then take the original JMP file with the data from the 
original DoE you used and do a least squares analysis on each response. When the window 
pops up, each response will have a little red triangle beside it. Under the triangle go to “Save 
Columns” and select “Predicted Formula”. You must save the formula NOT the values. JMP 
will create new columns in your table. IMPORTANT: Save this as a new file. Then import 
your random design into the design variable settings columns (ie the -1 and 1). JMP will 
automatically update the equations. These now are your predicted values for the random 
cases. You can then import your actual data from the random cases that you obtained from 
FLOPS/ALCCA and get the % relative error to these random cases where the relative error is: 
[((predicted-actual)/actual)*100]. You will observe that the error for these cases is MUCH 
bigger than for the original DoE and the R 2 values. This step gives you a fell for how good a 
representation your RSE’s are of your code at random points within your design space and 
plot with the distribution feature of JMP. Provide these distributions in your report 

8. Provide the general form of the RSE and also the specific coefficients that you have (in an 
appendix in tabular form, where the appendix should be entitled “Design Space RSE 
Coefficients”). What is the general assumption of 2 nd order RSE? How does that relate to the 
distributions you created in the previous requirement? Based on your answer, was a 2 nd order 
RSE appropriate? 

9. Create 2 contour plots (also called carpet plots or design plots) of TWR versus SW for your 
metrics and identify which metrics are constraining the design space based on the target and 
constraint values you identified in Step 1 for both the 10 and 25 year goals. 

10. If you do not have a feasible space for either year, how much would you have to relax the 
active constraint to obtain a feasible space? What would be the consequences of doing so in 
terms of competitiveness or regulatory compliance? 

11. Run a Monte Carlo Simulation on each RSE assuming a uniform distribution of all the design 
parameters. You have been provided with an MS Excel® spreadsheet which has a template 
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for the Crystal Ball macro of MS Excel®. Use 10,000 cases to generate the metric CDFs. 
Extract the results for the percentiles in 5% increments and use in MS Excel® to plot the 
results. What does the CDF represent? 

12. From the 6 th requirement in this list, determine an appropriate distribution for your error and 
then add that distribution to the original RSEs and rerun your Monte Carlo Simulation. 
Extract the data and compare the new CDFs to the CDFs that had no error term. What is the 
significance of the results 

Step 5: Determine System Feasibility and viability 

Once you have all your response CDFs, you need to determine how feasible and viable your design space 
is. If you have an unacceptable probability of success, you will need to infuse technology concepts in Step 
6. Additionally, once you determine how feasible your design space is, you will then investigate the 
economic viability. If you are not feasible or viable, this step will serve as a benchmark of how much 
improvement is needed. For the economic viability, use the economic variables below and the optimized 
configuration that you will determine in the second requirement of this step. 


Table IV Economic Variables for the Project 


Name 

ALCCA Variable Name 

ALCCA 

Namelist 

Variable 

Number 

Utilization 

U 

COPER 

1 

Production Quantity 

NV 

CMAN 

2 

Coach Load Factor 

| CLF 

COPER 

3 

First Class Load Factor 

FLF 

COPER 

4 

Airline Return on Investment 

| RTRTNA 

CMAN 

5 

Manufacturer Return on Investment 

| RTRTN 

CMAN 

6 

Fuel Cost 

| COFL 

COPER 

7 

Manufacturer’s Learning Curve 

| LEARN 1 

CMAN 

8 


LEARN2 


9 


LEARNA1 


10 


LEARNA2 


11 


LEARNAS1 


12 


LEARNAS2 


13 


LEARNFE1 


14 


LEARNFE2 


15 

Economic Range 

SL (second array element) 

COPER 

16 


Requirements for Step 5: 

Feasibility 

1. Take the CDF’s you generated in the previous step and add an additional vertical line for your 
target or constraint value for the metrics. Where that line intersects your CDF is the 
probability of achieving your goal. If every bit of the CDF lies to the left of your target, 100% 
of your designs can achieve your target. You are feasible. CONGRATS! Which CDF do you 
use? The one with the error term or without and why? Provide a table that lists the percent 
feasible space for each metric. 

2. Optimize your design variables with the desirability function in JMP, modify your FFOPS file 
(not the original, but a copy), and re-evaluate all of your metrics by running FFOPS with this 
new file. This will become your optimized configuration upon which you will apply the 
technology factors. Show a comparison of the new geometry to the original that was provided. 
Specifically, two tables, one for the change of the geometry settings and another for the 
resulting change in the metrics. Include three data columns for this latter table including the 
baseline metric value, the optimized baseline metric values, and the relative percent change of 
the optimized baseline values to the original baseline. This configuration will be used for the 
viability assessments below. 

3 . Identify some additional options that might make you vehicle feasible rather than just adding 
technologies. 
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Viability 

4. Define ranges for the economic variables and define the variables themselves 

5. Develop response surface equations (RSEs) for each economic metric from Table I as a 
function of the variables in Table IV for the optimized baseline you determined above. 

6. Create prediction profiles for these as a function of your variables. 

7. What is the R 2 for these RSE’S? 

8. Is there a residual? What does a residual mean? 

9. Define shape functions for each of the economic variables chosen within the Crystal Ball 
program. Run a Monte Carlo Simulation on your RSEs and extract the cumulative 
Distribution Functions (CDFs) and Probability Density Functions (PDFs) for each metric. 

10. Identify the probability of meeting your economic metric targets by adding in target or 
constraints values to the CDF’s 

1 1 . What is implied by economic uncertainty? 

12. Assume the following operational scenarios to modify your shape functions: 

1) Isolation: the U.S. is isolated to the world, no international travel is allowed, thus the 
economic mission, utilization, and load factor will be affected for the larger vehicles. 

2) No international oil available: the US must produce its own oil and thus fuel, this 
would drastically increase the price of fuel and also end up reducing the load factors 

3) Reduced Production: your competitor just launched a superior product that has resulted 
in you loosing production market share and thus you would need to reduce the production 
quantity and also the ROI to make the aircraft more economically appealing. 

4) Fabor Unions dissolve: all labor unions are abolished by the US and you loose your 
entire manufacturing capability, this would affect the associated learning curves 

5) Airline Re-regulation: government imposes new pricing regulations to the airlines and 
places fix prices on certain city pairs, this would affect the airline ROI and the load 
factors 

13. Compare the resulting CDFs and PDFs from the above scenarios to your baseline values from 
Step 3 (i.e., representative of today’s economics) and discuss the results. Provide the assumed 
shape functions, the significance of the simulation, and the results in your report. Discuss why 
you picked the shape distribution you did to model the different scenarios. 

Step 6: Specify Technology Alternatives 

The objective of this step is three fold. One, identify potential technologies that may improve technical 
feasibility and economic viability of your vehicle. Second, establish physical compatibility rules for the 
different technologies identified. Third, determine the impact, both improvements and degradations, to the 
system of interest. For the purpose of this project, you are provided the technologies as listed in Table V. 
Please use the nomenclature given for the technologies when you are providing your results. You have the 
freedom to add more technologies that you may know of or that you think would be appropriate. If you do 
this, please see Dr. Kirby for confirmation of your choices and the impact of the ones you selected. 

Next, you need to provide a compatibility matrix for the technologies. This matrix formalizes which 
technologies are physically compatible and as a by-product, reduces the number of alternatives to evaluate. 
This is important if the number of technologies considered for application is large and the combinatorial 
problem is out of hand. The compatibility matrix that you will use is shown in Figure 9. Note, a “1” implies 
the technologies are compatible and a “0” implies that a combination is NOT compatible. 

Finally, once the compatibility matrix is determined, the potential system and sub-system level impacts of 
each technology must be established. The impact should include primary benefits and secondary 
degradations to the entire system to be accurately assessed. This information is formalized in a Technology 
Impact Matrix (TIM). The TIM is the deterministic impact estimation for a technology that has been 
matured to the point of full-scale and widespread application. The TIM that you will use is shown in Figure 
10. Unfortunately, the technologies you were given did not have penalties. The techs were considered only 
from a benefit point of view. 


Page 19 





Exploration of Advanced Probabilistic and Stochastic Design Methods 


Table V: Subsonic Alternative Technologies 


ID# 

Technology Description 

Current 

TRL 

TRL=9 

Date 

1 

Adaptive Performance Optim. 

9 

2000 

2 

Stitched RFI Composite on Tail Skin 

4 

2006 

3 

Stitched RFI Composite on Tail Structure 

4 

2 66 6 

4 

Stitched RFI Composite on Wing Skin 

4 

2006 

5 

Stitched RFI Composite on Wing Structure 

4 

2006 

6 

Airframe Methods 

4 

2007 

7 

Fire Suppression 

3 

'2007 

8 

Low Cost Composite Manufacturing on Tail Structure 

2 

2009 

9 

Low Cost Composite Manufacturing on Wing Structure 

2 

2009 

10 

Propulsion System Health Management 

2 

2009 

11 

Smart Nacelle-PAI 

3 

2009 

12 

Emerging Alloy Tech & Forming on Tail Skin 

3 

2010 

13 

Emerging Alloy Tech & Forming on Tail Structure 

3 

2010 

14 

Emerging Alloy Tech & Forming on Wing Skin 

3 

2616 

15 

Emerging Alloy Tech & Forming on Wing Structure 

3 

2010 

16 

Superplastic Forming on Fuselage Skin 

2 

2611 

17 

Superplastic Forming on Tail Skin 

2 

2011 

18 

Superplastic Forming On Wing Skin 

2 

261 i 

19 

Russian Aluminum Lithium Fuselage Skin 

4 

2611 

20 

Adaptive Engine Control System 

4 

2011 

21 

Revolutionary Metallic Materials Systems on Fuselage Structure 

2 

2013 

22 

Revolutionary Metallic Materials Systems on Landing gear 

2 

2013 

23 

Revolutionary Metallic Materials Systems on Tail Structure 

2 

2013 

24 

Revolutionary Metallic Materials Systems on Wing Structure 

2 

2013 

25 

Composite Fuselage Shell (Fuselage Skin) 

2 

2013 

26 

Living Aircraft 

2 

2013 

27 

Active Load Alleviation on Tail 

4 

2013 

28 

Active Load Alleviation on Wing 

4 

2013 

29 

Antenna Systems 

2 

2014 

30 

Adaptive Wing Shaping 

3 

2014 

31 

Biologically Inspired Material Systems on Fuselage Structure 

1 

2015 

32 

Biologically Inspired Material Systems on Tail Structure 

1 

2015 

33 

Biologically Inspired Material Systems on Wing Structure 

1 

2015 

34 

BIOSANT on Fuselage Structure 

1 

2015 

35 

BIOSANT on Tail Structure 

1 

2015 

36 

BIOSANT on Wing Structure 

1 

2015 
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Figure 9: Subsonic Compatibility Matrix 
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Figure 10: Technology Impact Matrix 
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Requirements for Step 6: 

1. Describe in detail why the different technologies are used for your vehicle. Provide 
references. Note, some technologies are duplicates with applications for different components. 

Please do not repeat the description. 

2. Describe why each and every of the incompatible technologies are not physically compatible. 

Are the compatibility matrices correct? If not, explain why. 

3 . The values provided in the TIM assume that the technologies are all fully matured. What is an 
immature technology? Is the impact of an immature technology deterministic? If not, why 
isn’t it and would the impact values in the TIM be different from the ones provided? If so, 
describe why. What is the impact a function of? Discuss and justify your answers in detail. 
Provide any supporting references of your answers. 

4. Which research entities are working on the given technologies? What levels of research have 
been conducted thus far? A detailed description is required here with the addition of reference 
and source information. What is the rationale behind the impact values in the TIMs as applied 
to your vehicle? For example, why would the use of a composite wing reduce the cruise drag? 

5. From the descriptions you find, estimate the potential degradations to the system from 
integrating the technology to the full system. For example, the infusion of a composite wing 
could reduce the sized vehicle wing weight by 20% and the cruise drag (due to a smoother 
wing surface) by 2%. Yet, the costs associated with manufacturing and maintaining this type 
of wing will be more than a conventional aluminum wing structure due to increased 
complexity and material costs. This penalty can be simulated with increased Research, 
Development, Testing, and Evaluation (RDT&E), production (due to more expensive 
materials and processes), and Operation and Support (O&S) costs due to increased 
maintenance times required to repair damages and so on. Provide new impact values based on 
your discoveries to estimate the potential impacts. You can see Dr. Kirby for a little guidance 
or as a sanity check for the numbers you generate. 

6. Is the current level of research indicative of the TRL’s provided? Justify your answers. 

7. What is the significance of a TRL? Will the evaluation of the impact of the technologies on 
your vehicle be different if the technologies are a different TRL? If so, what must you 
consider? 

8. Provide the original compatibility matrix and the TIM in your report. 

9. Provide the modified TIM based on the results of requirement 5 above. 

10. Provide a series of TCMs for each year between 2006 and 2016 in an appendix with two in 
the main report being for the years 2007 and 2016 (indicative of the technologies available for 
2022). Call this appendix “Annual TCM”. This will result in 11 TCMs that you need to 
supply. The TCMs should be constructed based on the compatibility rules and which 
technologies have reach the TRL=9 date for the TCM of that year. 

Step 7: Assess Technology Alternatives 

You will be evaluating your technologies deterministically for this project. To do this, you need to create 
RSEs for the metrics defined in Table II as a function of the “k” factors (technology impact factors) listed 
in the TIM. You will do the RSEs on the optimal geometric configuration you determined in Step 5. The 
evaluation of the technologies identified in Step 6 will be performed deterministically. Hence, a full- 
factorial investigation is to be performed for a given year. For example, a full-factorial evaluation for nine 
technologies at two levels (i.e., “on” or “off’) constitutes 512 (2 n ~ 9 ) combinations and with eleven 
technologies is 2048. The technology evaluation is to be performed by creating a metamodel of each 
system metric as a function of the “k” vector elements. The metamodels are second-order Response Surface 
Equations (RSE) of the form: 

R = h o + YfAi + Z b a k ? + Z IW (1) 

i = 1 i = 1 i = 1 7=1+1 

where: R represents a given system metric; b t represent regression coefficients for linear terms; b u quadratic 
coefficients; by cross-product coefficients; k i? kj the “k” factor vector elements; and kjcj denotes interactions 
between two “k” vector elements. A metamodel, RSE, is created for each system metric via a Design of 
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Experiments (DoE) by bounding the “k” vector element ranges as shown in Table VI. The “0” implies no 
change in the technical metric if the reference point is “0” and a “1” imples no change if the reference point 
is “1” while a negative value denotes a reduction and a positive value an increase from the baseline values 
you have in your FLOPS/ALCCA input files. Once Eq (1) is determined for each metric, the RSEs can be 
used to rapidly evaluate the impact of the various technologies based on a particular “k” vector setting in 
lieu of executing FLOPS/ALCCA directly. 


Table VI: “K” Factors Ranges 


Generic Technology Impact Vector 

Minimum (%) 

Maximum (%) 

Wing Weight (skin or structure) 

0.65 

1.15 

Fuselage Weight (skin or structure) 

0.75 

i.o 

Hor Tail Weight (skin or structure) 

0.6 

l.o 

Ver Tail Weight (skin or structure) 

0.6 

1.0 

Cdi 

0.8 

i.o 

Cdo 

0.8 

1.0 

Landing Gear Weight 

0.75 

i.o 

Avionics Weight 

0.5 

1.05 

Hydraulics Weight 

0.5 

1.05 

Furnishing and Equipment Weight 

0.9 

1.05 

VT Area 

Same as design space 

Same as design space 

HT Area 

Same as design space 

Same as design space 

Engine Weight 

0.55 

1.05 

Fuel Consumption* 

0.8 

1.01 

RDT&E Costs* 

-20% 

+20% 

O&S Costs* 

-20% 

+20% 

Production Costs* 

-20% 

+20% 

Utilization* 

-20% 

+20% 

Wing Area* 

Same as design space 

Same as design space 

Thrust-to- weight Ratio* 

Same as design space 

Same as design space 


A full-factorial DoE should be created in JMP and the RSEs evaluated. As an example, the metric values 
obtained for an aircraft with aircraft morphing (T3) and IHPTET engines (T9) is depicted in Figure 11. A 

unique “k” vector is associated with each technology, specifically k 3 and k 9 . Since the impact of the 
various technologies is assumed to be additive, an alternative with T3 and T9 is simulated by adding each 
element of the vector resulting in a new vector, k 3+9 . The new vector is then fed into the RSEs and the 
metrics calculated. This is performed for every combination and every metric. 

One note: the maximum number of factors that JMP can consider for a full factorial is 13. 
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Case 


1 

2 

3 



T1 

-1 

-1 

-1 

1 


T2 

-1 

1 

-1 

1 


T3 

-1 

-1 

-1 



T9 Metric i Metric 2 Metric n 

~ # # UU # 

1 # # # 

1 # # # 


1 # # 



evaluations of Metric RSEs if all “1” implies technology applied Metric value is determined from the RSEs 

technologies are compatible 1 ” implies no technology 



Figure 1 1 : Example Technology Evaluation with "k" Vectors 


Requirements for Step 7: 

E Define the FLOPS/ALCCA variables, which correspond to the “k” factors given, and list the 
actual minimum and maximum of the variable. For example, if your baseline engine weight is 
1,000 lbs for a subsonic vehicle, then the actual minimum is 500 lbs (-50%) and the maximum 
is 1,100 lbs (+10%). 

2. How do you assess a given technology with the RSEs, i.e., how do you map the technology 
“k” vector to the RSEs? 

3. Provide the coefficients of the system level metric RSEs, the R 2 values, a discussion of 
representative whole model and residual plots, and the goodness of your RSEs. 

4. Provide a prediction profile for your metrics as a function of the “k” factors. See Figure 7 in 
Reference 5 for an example. 

5. Show contour plots (again TWR versus SW) of the most significant “k” factors and how they 
influence the feasible and viable space. What are the active constraints for a feasible and 
viable space? 

6. How much improvement is needed from the different technology impact factors to obtain a 
feasible space? To accomplish this, you can back solve your RSEs to determine the settings 
required to meet the metric targets. Provide 3 tables, one of the performance metrics and the 
required settings of the impact values to meet the metric targets independently, and one for 
the economics independently, and one of the settings to meet all target values concurrently. 
Discuss the differences of the values obtained. 

7. Provide a prediction profile for your metrics as a function of technologies for each year, ie 
2006 to 2016. In the main portion of the report, only put the 2007 and 2016 years. Put the 
remainder in another appendix called “Annual Technology Environments”. You will assume 
for this part that all technologies are compatible for that given year. See Figure 14 in 
Reference 3 for an example. The techs you will assess for each years will be: 
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Year 

Techs 

Number of Techs 

2006 

1,2, 3, 4, 5 

5 

2007 

1,2, 3, 4, 5, 6, 7 

7 

2008 

1,2, 3, 4, 5, 6, 7 

7 

now add on T1 for all, and call that your ’’NEW" baseline for 2009 

2009 

2,3,4,5,6,7,8,9,10,11 

10 

now add on T6, T7 for all and call that you ’’NEW #2” baseline for 2010 


(ie, NEW #2 should have T1,T6, T7) 



2010 

2,3,4,5,8,9,10,11,12,13,14,15 

12 

now merge the following technologies: 

T2 and T3 into T2-3 

T4 and T5 into T4-5 

T12 and T13 intoT12-13 

T45 and T15 intoT14-15 

2011 

T2-3 ,T4-5,T12- 13, TH- 
IS, 8, 9, 10,1 1,16,17,18,19,20 

12 

2012 

Same as 2011 


now add on T10, T1 1, T20, T27, and T28 for all, and call that your "NEW #3" baseline for 

2013 (ie, NEW #3 should have T1,T6, T7, T10, T1 1, T27, T28) 


2013 

T2-3,T4-5,8,9, 1 6, 1 7, 1 8,2 1 ,22,23 ,24,25,26 

13 

now add on T26 for all, and call that your "NEW #4" baseline for 2014 
(ie, NEW #4 should have T1,T6, T7, T10, T1 1, T26, T27, T28) 

2014 

T2-3,T4-5,8,9, 1 7, 1 8,2 1 ,22,23,24,25, 29, 30 

13 

now add on T29 for all, and call that your "NEW #5" baseline for 2014 
(ie, NEW #4 should have T1,T6, T7, T10, Til, T26, T27, T28, T29) 

2015 

25, 30,31,32,33,34,35,36 

8 

2016 

skip this one since it is the same as 2015 



8. Why did I pick the above technology combinations for a given year? Please rationalize and 
discuss other ways of assessing the technologies. 

9. Provide the shell scripts you used to run FLOPS/ALCCA and extraction of the data as an 
appendix called “Scripts”. 

10. Why are you evaluating the impact of the technologies on a fixed vehicle? Provide some 
rationale as to how you could investigate a technology space and a geometric design space 
concurrently? 

1 1 . How would you evaluate the impact of technologies in a probabilistic manner? 

12. In reality, are the technologies additive, where additive implies that the technology impacts 
are independent of each other. Discuss why or why not. If not, describe how you might 
evaluate a technology mix that is not additive. 

13. Create Decision Matrices for each year between 2006 and 2016 where the rows are the 
technology alternatives denoted as T1+T3+T32 or T5+T12+T19 and so on and the columns 
are the metrics. Put 2007 and 2016 in the main report and the others in an appendix entitled 
“Annual Decision Matrices”. 

14. If you were assessing your technologies in a probabilistic manner, how would you populate 
the matrix so that you would be comparing apples and apples? 
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Step 8: Select best family of alternatives 

This final step in the TIES method is where you select the “best” mix of technologies to satisfy the system 
level metrics. For any multi- attribute, -constraint, or -objective problem, the selection of the “best” family 
of alternatives is inherently subjective and no single answer will fulfill all requirements. Four approaches 
are proposed to account for the subjectivity of the problem: 

1. Multi- Attribute Decision-Making (MADM) techniques in the form of TOPSIS 

2. Technology Frontiers: Performance and Economic Effectiveness 

3. Technology Sensitivities: One-to-one technology comparison 

4. Genetic Algorithms (GA) 

Took at Reference 2 for a detailed description of how to do this aspect along with TIES for Dummies. One 
minor change from the reference is the change of the selection technique of “Resource Allocation” to 
“Technology Sensitivities” for clarity. 

You will be provided spreadsheets that will implement TOPSIS and the Frontiers. You may need 
to modify the spreadsheet for your particular problem. You will need to create a spreadsheet to calculate 
the sensitivities. 

Requirements for Step 8: 

1 . Describe how TOPSIS works. 

2. Consider at least 10 different weighting scenarios for TOPSIS and provide the top 10 
technology mixes that result for each year, again with 2007 and 2016 in the main report and 
the other in an Appendix entitled “Annual TOPSIS Scenarios”. Also, provide the weighting 
scenarios considered in tabular and graphical form. Discuss the significance of the results. Do 
any mix of technologies tend to appear in most of the different weighting scenarios? If so, 
discuss why. 

3. For the technology frontiers, provide the equations used to calculate the performance and 
economic frontiers and discuss what frontiers are. Establish threshold limits and discuss how 
they were established. 

4. Provide plots of PE versus EE for each year with 2007 and 2016 in the main report and the 
others in an appendix entitled “Annual Technology Frontiers”. The technology combinations 
should be grouped by how many technologies were on the vehicle at a given time, look at 
Figure 9 in Reference 2. Put the threshold limits on the plots and identify the feasible 
technology space and the “Ideal” and “Best Compromise” solutions, identify a few 
combinations as being compromises. If you do not have a clue how to do this in Excel, see Dr. 
Kirby for guidance. 

5. Describe the basic principles of how Genetic Algorithms work. 

6. Define the metrics which were the most sever from your feasibility study in Step 5. Based on 
these metrics consider 5 different Objective Functions and 5 different weighting scenarios for 
each Objective Function (25 total assessments) that will be used in a GA assessment to 
determine which technology sets improve these responses. Investigate the best technology 
sets that result from this investigation and determine which technology sets occur more often. 
From the GA assessment which technology combination is the best for each Objective 
Function and weighting scenario? What individual technologies appear in the resulting 25 top 
technology combinations? Comment on why these superior technologies are superior. If you 
were to pick one technology combination as the best which one would it be and why? 

7. Compare the best technology from the GA approach to the results you obtained from the 
TOPSIS approach and the frontier approach. Are the best technology sets from the different 
selection approaches the same? Discuss in detail these results of the different selection 
approaches. Which is the best technology set and rationalize your decision. 

8. For technology sensitivities, compare the influence of each individual technology to the 
baseline metrics you obtained from your geometric optimal design, i.e., what percent change 
from the baseline metrics do the technologies have on your baseline vehicle. Provide a plot 
for each metric. See Figure 16 in Reference 3 for an example. 
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9. Would the influence of the individual technologies (wrt technology sensitivities) change if 
you were assessing the impact probabilistically? If so, why? Qualitatively discuss what you 
think might happen. 

10. If a particular technology can help a metric, does it have a negative influence on other 
metrics? Discuss how each technology influences the metrics, whether positive or negative. 

11. If you were a program manager, which technologies would you invest development funds and 
why? Provide a summary of which technology combinations were superior from the different 
selection approaches to justify your decision. Provide tabular data, comparisons, or plots to 
justify. 

Closing the Loop: Reinvestigate the Design Space with Technologies 

The design of any complex, multi-attribute system is highly subjective, especially in the early phases of the 
development. Thus, the selection of a single concept alternative is highly dependent on the decision- 
maker’s judgement and relative importance of the evaluation criteria. Because of this, the family of 
alternative concepts that have been identified through the execution of TIES should be carried through the 
design process to retain design freedom as long as possible (think of the cost-knowledge-freedom curve). 
This process entails a re-investigation of the design space with the various technology alternatives that were 
deemed as the most significant from the selection step results. Subsequently, Steps 4 and 5 are repeated to 
determine if a different geometry will further increase the feasibility of the system for the family of 
technology alternatives. Thus, iteration is required within the TIES method. However, the iteration is rapid 
once the initial method is established. 

Requirements for Closure: 

1. Identify the best two technology combinations from Step 8, requirement 7 and provide a table 
of the technology combinations only for the 2016 year technologies. Provide a new alternative 
description scheme, i.e., for combo with T2+T5+T26, I will call that Alternative #1 and 
combo with T6+T17+T21, 1 will call that Alternative #2, and so on. 

2. Provide a table of the combined tech vector values (rows) against the technology 
combinations (columns). 

3. Modify your baseline FLOPS/ALCCA file to reflect each one of the alternatives that you 
established. Thus, you will have a baseline FLOPS/ALCCA file for Alt #1 and one for Alt #2. 

4. Provide a table of the comparisons of the metrics for the new alternative metrics to the 
baseline as you did with the optimized baseline, i.e., metric values and percent differences. 

5. Create RSEs of the metrics for each vehicle based on fixed “k” factors with the design 
variables in Table III. Check the goodness of your fit as you did in Step 4 but do not repeat 
the random cases or the viability assessments from Step 5. 

6. Extract the equations and repeat the Monte Carlo Simulation of the design space. Extract the 
percentiles again and compare these new CDF’s for each Alternative and metric to the 
original CDFs you generated in Step 5. Provide the CDF’s. 

7. Discuss the differences between the CDF’s you obtained. 

8. Provide a summary or a conclusion section to wrap up all of your ideas and efforts. 

Final Requirements 

1 . Discuss what you have learned and what is the value added to your knowledge through the 
implementation of the two projects. 

2. What difficulties did you encounter in the two projects? 

3. What difficulties did you encounter with FLOPS/ALCCA? 

4. What would you suggest for future design projects? 

5. Could you use the TIES method to design or evaluate other complex systems? 

6. What do you feel is missing in the TIES method? 

7. What other valuable feedback can you provide for future students? 
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Appendix A: JPDM Code and Application 

JPDM computer program implementation for PAV Application problem 

After the Data files are obtained, three simple steps should be taken to run the EDF model of JPDM. 

1 . Place these Data files in the same folder where the other MATLAB files are. The directory tree of 
the JPDM computer program is presented in the figure below. 

r a 

e 

e 

e 

0 

0 

0 

e 

e 

e 

e 

e 

e 

0 

b 

0 


2. Change the values of some parameters in the inputsEDF.m file for your specific problem. For 
example, 

maxvalue = [130 79 3.5]; 
minvalue =[0 0 0] ; 
crplots = [1 2]; 

name(l) = {'DOC[$/hr]} ; 
name(2) = {’NOISE [db]}; 


3. Run JPDM.m. Finally you will get the results including the probability table and 
corresponding graphs (Table 1 and Figure 4). For more details please refer to the Appendix 
D in reference 8. 
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MATLAB Scripts for JPDM PAV problem 

JPDM.m 


The first line in MATLAB® specifies the file name: 

% JPDM.m 

Initiation: erase memory 
clear all 

set clock to determine length of run for time comparisons 
ff = clock; 

Default Values 
for plotting only 
crplot s = [0 0 ] ; 
t = 1; 
sp = 0; 

aalow = [0 0 0 0 ] ; 
aahigh = [0 0 0 0]; 

calls the parameters specified in Input sEDF .m into the memory 
input sEDF 
%input s JPM 


determine the number of criteria (fom = figure of merit) from the number of maximum values specified in 
Inputs . m 

nfom = length (maxvalue) ; 


determine which method: EDF or JPM 
if method == 1 

determine number of alternatives from the number of specified distributions for the first criterion in 
Input s JPM . m 

points = length (distribution (: , 1 )) ; 

JPM 


elseif method == 2 

determine number of alternatives from the number of entries in the variable that specifies the size of the 
data sample in Datal, Data2, DataM 
points = length (1); 

EDF 

end 
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inputsEDF.m 

% inputsEDF.m 

specify method: JPM = 1, EDF 2 
method 2; 

definition of area of interest by assigning limits for each criterion 

maxvalue = [130 79 3.5]; 
minvalue = [000]; 

assigning number of steps taken for plotting; criteria in columns, alternatives in rows; different 
numbers of steps for different alternatives are possible 

nstep = [100 100 100; 100 100 100; 100 100 100]; 


defining length of data matrixes; 1 describes how many samples were collected for each 
alternative (number can vary with alternative) 

\= [ 10000 ; 10000 ; 10000 ]; 


assignment of which criteria are plotted against each other (here Criterion 1 against Criterion 
2); can be expanded to many more rows, limited to two columns 
crplots = [12]; 

assigning a range for each criterion that the joint distribution is conditioned on (here 1 and 2); 
criteria are in columns, minimum in first row, maximum in second row; assigning ranges to 
criteria identified for plotting in crplots has no effect on that particular plot. 
condrange = [-100 -100; 100 100]; 

plotting identifier for joint distributions in 2-D (density), 3-D (density), and cumulative 

probability (2-D); y =yes, n = no 

probplot2D = f y ? ; 

probplot3D = f n ? ; 

cumplot = ? n’; 


identify values of joint density to be plotted in 2-D (‘cuts ’ through joint probability hump); major 
parameter for creating desired plots 
cp = (0.00005:0.0003:0.05); 


identify values of joint cumulative probability to be plotted in 2-D 
cc = [0,0.1, .2, .3, .4, .5, .6, .7, .8, .9]; 

identify names for criteria to be plotted 
name(l) = {T>OC[$/hr] ? }; 
name(2) = { ? NOISE[db] ? }; 


identify colors for each alternative distribution (refer to MATLAB manual for color codes) 
contourcolors = [ ? k ? ; 'b f ; V]; 
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EDF.m 

% EDF.m 

dat = NaN * ones (nfom,max(l), points); 

fid(l) = fopen(’Datal',V); 

fid(2) = fopen(’Data2',V); 

fid(3) = fopen(’Data3’,V); 

fid(4) = fopen(’Data4’,’r’); 

fid(5) = fopen(’Data5’,V); 

fid(6) = fopen('Data67r'); 

fid(7) = fopen(’Data7’,V); 

fid(8) = fopen(’Data8’,V); 

fid(9) = fopen(’Data9’,V); 

fid(10) = fopen(’Datal07r'); 

fid(l 1) = fopen(’Datal E,’r'); 

for n = 1 :points 

dat(:,:,n) = fscanf(fid(n),’%g’,[nfom l(n)]); 
data(:,:,n) = dat(:,:,n)'; 
end 

status = fclose('aH’); 
for n = 1 :points 

xmin(n,:) = min(data(:,:,n),[],l); 
xmax(n,:) = max(data(:,:,n),[],l); 
lower(n,:) = max(xmin(n,:),minvalue); 
upper(n,:) = min(xmax(n,:),maxvalue); 
fidd = fopen('passEDF7w'); 
if n == 1 

leopy Datal Data 
elseif n == 2 
!copy Data2 Data 
elseif n == 3 
!copy Data3 Data 
elseif n == 4 
!copy Data4 Data 
elseif n == 5 
!copy Data5 Data 
elseif n == 6 
leopy Data6 Data 
elseif n == 7 
leopy Data7 Data 
elseif n == 8 
leopy Data8 Data 
elseif n == 9 
leopy Data9 Data 
elseif n == 10 
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!copy Data 10 Data 
elseif n == 1 1 
!copy Datal 1 Data 
end 

if nfom = 2 

step(n,:) = (xmax(n,:)-xmin(n,:))./nstep(n,:); 

fprintf(fidd ,'%g\n' ,l(n) ,xmin(n, : ) ,xmax(n, : ) , step(n, : ) ,lo wer(n, : ) ,upper(n, : )) ; 
status = fclose(fidd); 

IEDF2.exe 

x(n,:) = xmin(n,l) + step(n,l)/2:step(n,l):xmax(n,l); 
y(n,:) = xmin(n,2) + step(n,2)/2:step(n,2):xmax(n,2); 
fidd = fopen(’passback’,V); 
i = fscanf(fidd,’%g’,[l 1]); 
j = fscanf(fidd/%g’,[l 1]); 
count = fscanf(fidd,'%g',[i j]); 

C(n) = fscanf(fidd,'%g',[l 1]); 
countv(n,:) = fscanf(fidd,'%g',[l nfom]); 

status = fclose(fidd); 
prob(:,:,n) = count(2:i,2:j)/l(n); 

prob(l,:,n) = (count(l,2:j) + count(2,2:j))/l(n); 
prob(:,l,n) = (count(2:i,l) + count(2:i,2))/l(n); 

plotx = 1 ; 
ploty = 2; 
if n == points 
cd plot.files 

if probplot2D — y 
contourplot 
end 

if probplot3D — y 
surfplot 
end 

if cumplot = y 
cumprobplot 

end 
cd .. 
end 

elseif nfom == 3 

fprintf(fidd,'%g\n',l(n),lower(n,:),upper(n,:)); 
status = fclose(fidd); 

IEDF3.exe 

fidd = fopen(’passbackVr'); 

C(n) = fscanf(fidd;%g’,[l 1]); 
countv(n,:) = fscanf(fidd,'%g',[l nfom]); 

status = fclose(fidd); 
elseif nfom == 4 

fprintf(fidd,'%g\n’,l(n),lower(n,:),upper(n,:)); 
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status = fclose(fidd); 

IEDF4.exe 

fidd = fopen(’passbackVr'); 

C(n) = fscanf(fidd/%g',[l 1]); 
countv(n,:) = fscanf(fidd,'%g’,[l nfom]); 

status = fclose(fidd); 
elseif nfom == 5 

fprintf(fidd,'%g\n',l(n),lower(n,:),upper(n,:)); 
status = fclose(fidd); 

IEDF5.exe 

fidd = fopen(’passbackVr’); 

C(n) = fscanf(fidd;%g’,[l 1]); 
countv(n,:) = fscanf(fidd,'%g',[l nfom]); 
status = fclose(fidd); 
end 
end 

if sum(sum(crplots)) > 3 
if nfom == 3 
elseif nfom == 4 
crosspl4 

elseif nfom == 5 
end 

ConditionalProbability(3:points+2,l) = [l:points]'; 
for m =l:length(crplots(:,l)) 

ConditionalProbability(l:2,m+l) = [crplots(m,l);crplots(m,2)]; 
ConditionalProbability(3:points+2,m+l) = ccond(:,m)./cl(:,m); 
end 

ConditionalProbability 

end 

cumulative = C'./l; 

probtable = zeros(points+l,nfom+2); 
probtable(2:points+l,l) = transpose^ :points); 
probtable(l,2:nfom+2) = (0:nfom); 
probtable(2:points+l,2) = cumulative; 
for n = 1 :points 

probtable(n+l,3*nfom+2) = countv(n,:)/l(n); 
end 

probtable 

if sum(sum(crplots)) > 3 
cl 
end 

ee = etime(clock,ff); 
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EDF2.exe 

program EDF2 


implicit none 




real 

x. 

y, xmin. 

xmaxj 

, ymin. 

ymax, stepx 

real 

minx, maxx, 

miny , 

r maxy. 

data (15000, 2) 

real 

count ( 1001 , 

1001) , 

r stepy. 

epsx, epsy 

integer 

k, i, j. 

1, s, 

r t, C, 

CX, CY 

o 

ll 

o 






CX = 

0 





CY = 

0 





open 

(3 

, FILE = 

' passEDF ' , status = ’old’) 

read 

(8 

, *) 1, minx, miny, maxx, maxy, stepx 

read 

(8 

, *) stepy, xmin, ymin 

., xmax, ymax 

open 

(9 

, FILE = 

’ Data 

’ , status = ' old ’ ) 

epsx 

= 

stepx/100 




epsy 

= 

stepy/100 




do 10 

k 

i — 1 
x — 1 

ll 





read (9, *) data(k,l), data (k, 2) 
j = 0 

do 20 x = minx,maxx+epsx, stepx 
i = 0 
j = j + i 

do 30 y = miny , maxy+epsy , stepy 
i = i + 1 

if (data(k,l) . gt . x - stepx) then 
if (data(k,l) . le . x) then 

if (data (k, 2) . gt . y - stepy) then 

if (data (k, 2) .le. y) then 
count (i,j) = count (i,j) + 1 
endif 
endif 
endif 
endif 

30 continue 

20 continue 

if (data(k,l) . ge . xmin) then 
if (data(k,l) .le. xmax) then 
if (data (k, 2) . ge . ymin) then 

if (data (k, 2) .le. ymax) then 
C = C + 1 
endif 
endif 
endif 
endif 

if (data(k,l) . ge . xmin) then 
if (data(k,l) .le. xmax) then 
CX = CX + 1 
endif 
endif 

if (data (k, 2) . ge . ymin) then 

if (data (k, 2) .le. ymax) then 
CY = CY + 1 
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endif 

endif 

10 continue 

open (10, FILE = 'passback') 
write (10, *) i, j 

do 40 s = 1, j 
do 50 t = 1 , i 

write (10, *) count (t, s) 

50 continue 

40 continue 

write (10, *) C 
write (10, *) CX 
write (10, *) CY 
endfile (10) 
close (10) 
close (9) 
close ( 8 ) 
end 

EDF5.exe 

program EDF5 

implicit none 

real min(5), max (5), data ( 15000 , 5 ) 
integer k, 1, s, D, C(5) 

D = 0 

do 5 s =1,5 
C (s) =0 
5 continue 

open (8, FILE = 'passEDF', status = 'old') 


read 

(8, *) 1, (min (s) , s : 

LO 

\ — 1 
II 

(max ( s ) , s=l , 5 ) 



open 

(9, FILE = 'Data', status = 

= 'old') 



do 10 

k = 1,1 





read (9, *) (data(k,s). 

LO 

T 1 

II 

CO 




if 

(data(k,l) .ge.min(l) 

. and . 

data(k,l) . le.max 

(i) > 

then 

if 

(data (k, 2) .ge.min(2) 

. and . 

data (k, 2) .le.max 

(2) ) 

then 

if 

(data (k, 3) .ge.min(3) 

. and . 

data (k, 3) .le.max 

(3) ) 

then 

if 

(data (k, 4) .ge.min(4) 

. and . 

data (k, 4) .le.max 

(4) ) 

then 

if 

(data (k, 5) .ge.min(5) 

. and . 

data (k, 5) .le.max 

(5) ) 

then 


D=D + 1 


endif 

endif 

endif 

endif 

endif 

do 20 s = 1,5 

if (data (k, s ) . ge . min ( s ) .and. data (k, s ). le . max ( s )) then 
C (s) =C (s) +1 
endif 

20 continue 

10 continue 
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open (10, FILE = ' passbackEDF ' ) 
write (10, *) D 
do 40 s = 1,5 

write ( 10 , * ) C ( s ) 

40 continue 

endfile (10) 
close (10) 
close (9) 
close ( 8 ) 

end 


contourplot.m 

Objective: Creates a contour plot of the joint probability function for two criteria. 

% contourplot.m % 
figure 

hold figure to overlay area of interest, distribution, and limits ; plot is created for two criteria at a time: 

plotx and ploty, assigned in EDF. m or JPM. m 

hold 


create grid over area of interest, identified by green “ dots ’’plotted at grid points 

[xm, ym] = meshgrid (max (min (min (x) ) , minvalue (plotx) ) tstep ( 1 , plotx) 

/2 : step ( 1 , plotx) : min (max (max (x) ) , maxvalue (plotx) ) - 
step ( 1 , plotx) / 2 , max (min (min (y ) ) tstep ( 1 , ploty ) / 2 , 
minvalue (ploty) ) : step ( 1 , ploty ) : min (max (max (y ) ) , 
maxvalue (ploty) ) -step ( 1 , ploty ) / 2) ; 

plot (xm, ym, ' g . ’ ) ; 

plot distribution for each alternative 
for n = 1 : points 

plot contours of constant joint probability (prob) over the first (xm) and second (ym) criterion, cp is 
vector with probability values to be plotted 

[Cont,H] = contour (x (n, :), y (n, :), prob (:,:, n) , cp, '-') ; 
set (H, ’Color ! , contour colors (n, : ) ) 

end 


plot limits of area of interest 

assigns values which the limits are plotted against 

yep = min (min (x) ) : step ( 1 , plotx) : max (max (x) ) ; 

xcp = min (min (y) ): step ( 1 , ploty ): max (max (y) ) ; 


identifies whether minimum limit for first criterion (x) needs to be plotted 
if minvalue (plotx) > min (min (x) ) 

assigns limit values to vector 

lo = ones ( 1 , length (xcp) ) *minvalue (plotx) ; 

plots line by plotting constant x-value against changing y -values (x-limit) 
plot ( lo, xcp) 
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clear lo 
end 


identifies whether minimum limit for second criterion (y) needs to be plotted 
if minvalue (ploty) > min (min (y) ) 

assigns limit values to vector 

lo = ones ( 1 , length (yep) ) *minvalue (ploty ) ; 

plots line by plotting constant y-value against changing x-values (y-limit) 
plot (yep, lo) 

clear lo 
end 


identifies whether maximum limit for first criterion (x) needs to be plotted 
if maxvalue (plotx) < max (max (x) ) 

assigns limit values to vector 

lo = ones ( 1 , length (xcp) ) *maxvalue (plotx) ; 

plots line by plotting constant x-value against changing y -values (x-limit) 
plot ( lo, xcp) 
clear lo 
end 


identifies whether maximum limit for second criterion (y) needs to be plotted 
if maxvalue (ploty) < max (max (y) ) 

assigns limit values to vector 

lo = ones ( 1 , length (yep) ) *maxvalue (ploty ) ; 

plots line by plotting constant y-value against changing x-values (y-limit) 
plot (yep, lo) 

clear lo 
end 


focus plot on area just plotted or on area specified in inputs .m 
if sum(aalow) = 0 

axis tight 
else 

axis ([ aalow (plotx) aahigh (plotx) aalow (ploty) aahigh (ploty )] ) 
end 

label x- and y-axes 
xlabel (name (plotx) ) ; 
ylabel (name (ploty) ) ; 

save plot in file named ‘prob2D.jpg’ 

print -djpeg95 prob2D.jpg 
hold 


The contour plot displays lines of constant probability that indicate in two dimensions in which 
region the joint probability distribution resides. The plot also indicates the area of interest, i.e. values that 
satisfy both figures of merit. Hence, the plot symbolizes the probability of satisfying the FOMs by the area 
within probability lines, which fall within the area of interest. If the lines of constant probability fall well 
within the area of interest, the probability is large. If they fall outside, the probability is small. 

The plot is further an excellent visual comparison of the different products in the study. The lines 
of constant probability of all products are overlaid and can be compared by inspection as to how well they 
satisfy the individual or joint figures of merit. 
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TOFL 

Contour plot from JPM method 



surfplot.m 


Objective: Creates a surface (JPM) or column (EDF) plot, of the joint probability function for two 
criteria. 


% surfplot.m 
figure 


identify whether all cases are supposes to be overlaid. 
if sp == 0 

plots are created for each case and overlaid. 
for n = 1 : points 

for method 2 (EDF) a column plot is more desirable. 
if method == 2 

for the column plot the order of columns needs to be reversed. 
tprob ( : , : , n) = f liplr (prob ( : , : , n) ) ; 

print 3D column plot. Note : the function ‘bar3’ does not label the x-axis in the current version. 

Mathworks promised to change that in one of the next versions ofMatlab. 
graph = bar3 (y (n, : ) , tprob ( : , : , n) ) ; 

change view angle of the plot. 
view (155,20) ; 

else 

create continuos surface plot for joint density from JPM method. 

graph = surf (x (n, :), y (n, :), prob n) ) ; 

change view angle of the plot. 
view (-15, 30 ) ; 
end 

change line color to light gray for better printing in black-and-white. 
set (graph, ! EdgeColor ’ , [ 0 . 6 0.6 0.6]); 
hold on 
end 
else 

only one plot is created, sp is identifier as to which one of the products plotted. 
if method == 2 
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tprob = f liplr (prob ( : , : , sp) ) ; 
graph = bar3 (y ( sp, :), tprob) ; 
view (155,20) ; 
else 

graph = surf (x ( sp, :), y ( sp, :), prob (:,:, sp) ) ; 
view (-15, 30 ) ; 
end 

set (graph, ' EdgeColor ' , [ 0 . 6 0.6 0.6]); 

hold on to existing plot. 

hold on 
end 


focus plot on area just plotted 
axis tight 

label x- and y- and z-axes 
xlabel (name (plotx) ) ; 
ylabel (name (ploty) ) ; 
zlabel ( 1 Density 1 ) ; 

save plot in file named ‘prob2D.jpg’ 

print -djpeg95 prob3D.jpg 




Surface plot from JPM method 


Column plot from EDF method 


Note : the function ‘bar 3 ’ does not label the x-axis in the current version. Mathworks promised to change 
that in one of the next versions of Matlab. 


Page 39 


Exploration of Advanced Probabilistic and Stochastic Design Methods 


cumprobplot.m 

% cumprobplot.m % 

set step size for both FOMs to one if Method 2 (EDF) was used to calculate prob. EDF counts 
occurrences of data, i.e. mass. Thus, the cumulative probability, calculated by summation not integration 
of prob, does not need to be normalized by the step size. 
if method == 2 

step = ones (point s , nfom) ; 
end 

determine size of prob. 

[ey, ex] = size (prob ( : , : , 1) ) ; 

for n = 1 : points 

reduce size of vector for second criterion. This reduces the runtime for a fine grid. The cumulative 
probability plot can afford to have a larger grid than the joint density plot. 
for i = 1: round (t) : ey 

kt = 1+ (i-1) /round (t) ; 

reduce size of vector for first criterion. 

for j = 1: round (t) : ex 

it = 1+ ( j-1) /round (t) ; 

sum/numerically integrate probability values over area of interest. 
if direction (plotx) == '1' 

if direction (ploty) == '1' 

cmp(kt,lt,n) = sum ( sum (prob ( i : ey , j : ex, n) ) ) ; 
elseif direction (ploty) == 's' 

cmp(kt,lt,n) = sum (sum (prob ( 1 : i, j : ex, n) ) ) ; 

end 

elseif direction (plotx) == ’s’ 
if direction (ploty) == ’1’ 

cmp(kt,lt,n) = sum ( sum (prob ( i : ey, 1 : j , n) )) ; 
elseif direction (ploty) == 's' 

cmp(kt,lt,n) = sum (sum (prob ( 1 : i, 1 : j , n) )) ; 

end 

end 

end 

end 

cump(:,:,n) = cmp ( : , : , n) *prod ( step (n, : ) ) ; 

end 

figure 

for n = 1 : points 

plot contours of constant cumulative probability (cump) over the first (xc) and second (yc) FOM. cc is 
vector with probability values for plot. 

[Cont, H] = contour (x (n, : ) , y (n , : ) , cump ( : , : , n) , cc, ' - ' ) ; 
set (H, 'Color ' , cont our colors (n, : ) ) 

c label (Cont, H, ' font size ' , 8, 'color ' , cont our colors (n, : ) , ' label spacing ' , 

1000) ; 

hold on to plot in order to overlay all products that need to be compared. 
hold on 

end 

plot crossbar for values of interest. 
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if maxvalue (plotx) > xmax(n,plotx) 

cx = ones ( 1 , length (y (n, :))) *minvalue (plotx) ; 
if maxvalue (ploty) > xmax (n, ploty ) 

cy = ones (1, length (x (n, : ) ) ) *minvalue (ploty) ; 
elseif minvalue (ploty) < xmin (n, ploty ) 

cy = ones (1, length (x (n, : ) ) ) *maxvalue (ploty) ; 
end 

elseif minvalue (plotx) < xmin (n, plotx) 

cx = ones (1, length (y (n, : ) ) ) ^maxvalue (plotx) ; 
if maxvalue (ploty) > xmax (n, ploty ) 

cy = ones (1, length (x (n, : ) ) ) *minvalue (ploty) ; 
elseif minvalue (ploty) < xmin (n, ploty ) 

cy = ones (1, length (x (n, : ) ) ) ^maxvalue (ploty) ; 
end 
end 

plot (x, cy) 
plot ( cx, y ) 


focus plot on area just plotted or on area specified in inputs .m 
if sum(aalow) = 0 

axis tight 
else 

axis ( [aalow (plotx) aahigh (plotx) aalow (ploty) aahigh (ploty )] ) 
end 

label x- and y-axes. 
xlabel (name (plotx) ) ; 
ylabel (name (ploty) ) ; 

save plot in file named ‘cum2D.jpg’. 

print -djpeg95 cum2D.jpg 

The cumulative probability plot displays lines of constant cumulative probability. The cumulative 
probability captures all values in the area of interest already. Hence, in this plot there is no area but a point 
of interest, indicated by the cross mark. In addition to reading the probability value off the lines at the 
mark, the plot facilitates two major investigations. First, a comparison of products that uses an overlay of 
all products’ cumulative probability plots, indicating which product yields the highest probability. Second 
and more importantly, it can be used for ‘What-if-studies’. For example, the plot would immediately tell 
what the new probability value is, if the values of interest were to change. It can demonstrate directly how 
one product that may not have had a high probability for the first set of values performs better for the 
second, while another one performs conversely. The plot facilitates another type of ‘What-if-studies’: 
“How much can I gain in one criterion, if I was able to relax the other?” These trade-offs can be done in 
the knees of the lines. While holding the probability constant, i.e. at a constant chance of achieving the 
criterion values, one criterion can be traded against the other. These trades then can be done at different 
probability levels, thus indicating the change in trade with changing probability. 
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Cumulative probability plot (JPM method) Cumulative probability plot (EDF method) 


Appendix B: Representative TIES Implementation 

Please see the electronic file entitled “150 Pax Report Final.doc” or “150 Pax Report Final.pdf’ on the CD. 

Appendix C: TIES for Dummies Tutorial 

Please see the electronic file entitled “TIES_for_Dummies_JMP_Ver4.pdf’ in the New TIES Demo” 
directory of the CD. 
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